Compression enables storing, transmitting, and processing of videos with fewer storage, network, and processor resources. The most widely used video compression standards include MPEG-1 for storage and retrieval of moving pictures, MPEG-2 for digital television, and H.263 for video conferencing, see ISO/IEC 11172-2:1993, “Information Technology—Coding of Moving Pictures and Associated Audio for Digital Storage Media up to about 1.5 Mbit/s—Part 2: Video,” D. LeGall, “MPEG: A Video Compression Standard for Multimedia Applications,” Communications of the ACM, Vol. 34, No. 4, pp. 46-58, 1991, ISO/IEC 13818-2:1996, “Information Technology—Generic Coding of Moving Pictures and Associated Audio Information—Part 2: Video,” 1994, ITU-T SG XV, DRAFT H.263, “Video Coding for Low Bitrate Communication,” 1996, and ITU-T SG XVI, DRAFT13 H.263+Q15-A-60 rev.0, “Video Coding for Low Bitrate Communication,” 1997.
These standards are relatively low-level specifications that deal primarily with spatial compression of images or frames, and the spatial and temporal compression of a sequence of frames. As a common feature, these standards perform compression on a per frame basis. These standards achieve high compression ratios for a wide range of applications.
Video coding standards, such as MPEG-4 for multimedia applications provide several new coding tools, including tools to improve the coding efficiency, and tools that support object-based coding and error-resilience, see ISO/IEC 14496-2:1999, “Information technology—coding of audio/visual objects, Part 2: Visual.”
One of the main problems in delivering video content over networks is adapting the content to meet particular constraints imposed by users, networks and devices. Users require playback with minimal variation in perceived quality. However, dynamic network conditions often make this difficult to achieve.
Fine granular scalable (FGS) coding has been adopted by the MPEG-4 standard. The tools that support FGS coding are specified in an amendment of the MPEG-4 standard, “ISO/IEC 14496-2:1999/FDAM4, “Information technology—coding of audio/visual objects, Part 2: Visual.” An overview of FGS coding is described by Li in “Overview of Fine Granularity Scalability in MPEG-4 video standard,” IEEE Trans. Circuits and Systems for Video Technology, March 2001.
FGS coding is a radical departure from traditional scalable coding. With traditional scalable coding, the video is coded into a base layer bitstream and possibly several enhancement layer bitstreams, where the granularity is only as fine as the number of enhancement layer bitstreams that are formed. The resulting rate-distortion (R-D) curve resembles a step-like function.
In contrast, FGS coding provides an enhancement layer bitstream that is continuously scalable. Providing a continuous scalable enhancement layer bitstream is accomplished by a bit-plane coding method that uses discrete cosine transform (DCT) coefficients. Bit-plane coding allows the enhancement layer bitstream to be truncated at any point. In that way, the quality of the reconstructed video is proportional to the number of decoded bits of the enhancement layer bitstream.
FIG. 1 shows a conventional FGS encoder including a base layer encoder 102 and an enhancement layer encoder 101. An input video 103 is provided to a typical base layer encoder. The base layer encoder includes DCT 110, Quantization (Q) 120, motion compensation (MC) 130, inverse quantization (Q−1) 140, inverse DCT (IDCT) 150, motion estimation 160 with motion vectors 161, clipping 170, frame memory 180, and variable length coder (VLC) 190 components. The output of the base layer encoder is a base layer bitstream 104 having some predetermined minimum constant bit-rate (CBR). Typically, the CBR is very low, for example, 20 Kbps or less. Thus, the base layer bitstream 104 can be transmitted over high and low bandwidth channels.
An enhancement layer bitstream 105 is generated by subtracting reconstructed frames of the base layer bitstream 103 from the input video. This yields an FGS residual signal in the spatial domain. Enhancement layer encoding is then applied to the residual signal. The enhancement encoder 101 includes a DCT 190, followed by bit-plane shifting 192, a maximum operation 194, and bit-plane VLC coding 196 to produce the enhancement layer bitstream 105.
FIG. 2 shows a FGS decoder including a base layer decoder 201 and an enhancement layer decoder 202 that are applied respectively to the base layer bitstream 104 and the enhancement layer bitstream 105 to produce a reconstructed a base layer bitstream 291 and a reconstructed enhancement layer bitstream 292. The decoder includes a variable length decoder (VLD) 210, inverse quantizer 220, inverse DCT 230, motion compensation 240, frame memory 250. The FGS residual signal 292 is reconstructed by passing the enhancement layer bitstream 105 through bit-plane VLD 260, bit-plane shift 270 and IDCT 280 components. The FGS residual signal 292 can then be added to the reconstructed base layer bitstream 291 to yield the enhancement video 293. The combined signal 293 is clipped to ensure that the signal is bounded, i.e., 8-bit pixels values are in the range [0, 255].
A selective enhancement method to control the bit-plane shifting in the enhancement layer bitstream of the FGS coded video bitstream was described in U.S. Pat. No. 6,263,022, “System and method for fine granular scalable video with selective quality enhancement,” issued on Jul. 17, 2001 to Chen, et al. There, the quantization parameter used for coding the base layer video also determined the corresponding shifting factor. The bit-planes associated with macroblocks that were deemed more visually important were shifted higher.
A key point to note is that the bit rate of the base layer bitstream is some predetermined minimum. The enhancement layer bitstream covers the range of rates and distortions from a minimum to near lossless reconstruction. Also, after the enhancement layer bitstream has been generated, it can be stored and re-used many times. According to e.g., network characteristics, an appropriate number of bits can be allocated to a frame and transmitted over the network, taking into consideration current network conditions. It is important to note however that there is no quantization parameter to adjust in that scheme.
The MPEG-4 standard does not specify how rate allocation, or equivalently, the truncation of bits on a per frame basis is to be done. The standard only specifies how the scalable bitstream is decoded. Additionally, traditional methods that have been used to model the rate-distortion characteristics, e.g., methods based on quantization parameters, no longer hold with a bit-plane coding scheme used by the FGS coding. As a result the perceived quality of the reconstructed video can vary noticeably.
Because differential sensitivity is key to human visual perception, it is important to minimize the variation in the perceived quality rather than overall distortion. Optimal rate allocation can be done by minimizing a cost according to an exponential R-D model. This leads to a constant quality among the decoded frames, see Wang, et al., “A new rate allocation scheme for progressive fine granular scalable coding,” Proc. International Symposium on Circuits and Systems, 2001. However, the prior art rate allocation methods typically use exhaustive searches not suitable for real-time applications, and do not work on low bit-rate signals. In U.S. patent application Ser. No. 09/961,987, “Transcoder for Scalable Multi-Layer Constant Quality Video Bitstreams,” filed on Sep. 24, 2001 by Zhang, et al., an FGS-based transcoder that extracts R-D labeling points has been described to provide an output bitstream that has a constant quality.
The FGS coding method and rate allocation techniques described above are useful for transmission over dynamic channels. The key assumption made is that the receiving device has a decoder that can process both base and enhancement layer bitstreams. In practice, this may not always be true, especially for today's low-power mobile devices, such as cellular telephones and personal digital assistants (PDAs).
The only existing way to overcome this problem is to simply transmit the base layer bitstream to the receiving device. The main drawback of that approach is that the base layer bitstream is usually coded with a minimum constant bit-rate resulting in a very low quality decoded video. Therefore, if there is additional bandwidth available, then the connection and the device capabilities are under-utilized. In order to convert the FGS coded video to a single layer bitstream with higher quality than the base layer bitstream only, some other means of transcoding is required.
As shown in FIG. 3a, this can be accomplished by a transcoder 300. In a simplest implementation, the transcoder includes an FGS decoder, including a base layer decoder 320, an enhancement layer decoder 310, an adder 330, and a single layer encoder 340. An input compressed base layer bitstream 301 and a compressed FGS enhancement layer bitstream 302 are fully decoded, then re-encoded to a single layer bitstream 303 with output bit-rate Rout. Usually, the output rate is lower than the input rate. In practice, full decoding and full encoding in a transcoder is not done due to the high complexity of encoding the decoded bitstream. The detailed operation of this transcoder 300 is shown in FIG. 3b where the components function as described above for FIGS. 1 and 2.
A prior art method for transcoding and scaling a bitstream has been described by Sun et al., in “Architectures for MPEG compressed bitstream scaling,” IEEE Transactions on Circuits and Systems for Video Technology, April 1996. There, four methods of rate reduction, with varying complexity and architecture, were described.
FIG. 4 shows one example method. That method is referred to as a single layer, closed-loop architecture. In that method, an input video bitstream 401 is partially decoded, i.e., macroblocks of the input bitstream are variable-length decoded (VLD) 410, and inverse quantized with Q1 420 to yield discrete cosine transform (DCT) coefficients. Correction DCT coefficients are added 440 to the incoming DCT coefficients to compensate for the mismatch produced by re-quantization. That correction improves the quality of the reference frames that will eventually be used for decoding. After the correction has been added, the newly formed blocks are re-quantized with Q2 450 to satisfy a new rate, and variable-length coded 460.
To obtain the correction DCT coefficients 430, the re-quantized DCT coefficients are inverse quantized 470 and subtracted 480 from the original partially decoded DCT coefficients. That difference is transformed to the spatial domain via an inverse DCT (IDCT) 490 and stored into a frame memory 495. The motion vectors associated with each incoming block are then used to recall the corresponding difference blocks, such as in motion compensation 496. The corresponding blocks are then transformed via the DCT 430 to yield the correction component. A derivation of the method shown in FIG. 4 is described in “A frequency domain video transcoder for dynamic bit-rate reduction of MPEG-2 bitstreams,” by Assuncao et al., IEEE Transactions on Circuits and Systems for Video Technology, pp. 953-957, 1998.
Assuncao et al. also described an alternate method for the same task. In the alternative method, they used a motion compensation (MC) loop operating in the frequency domain for drift compensation. Approximate matrices were derived for fast computation of the MC blocks in the frequency domain. A Lagrangian optimization was used to calculate the best quantizer scales for transcoding. That alternative method removed the need for the IDCT/DCT components.
The prior art clearly teaches methods of transcoding compressed single layer bitstreams for bit-rate reduction. However, the prior art does not teach a method of transcoding a multi-layer bitstream to a single layer stream. In particular, there is a need for efficient transcoding methods that convert the multi-layer bitstream consisting of a base and FGS enhancement layers to a single layer bitstream with a higher quality than the base layer bitstream.